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1.  OBJECTIVES 

The  original  objectives  of  the  grants  were: 

•  Investigate  the  design  and  fabrieation  of  a  novel  eustom  made  imaging  sensor 
with  foeal  plane  speetral  detection  capabilities. 

•  Modeling  of  the  electromagnetic  wave  interaction  with  aluminum  nanowires 
structures  to  examine  polarization  properties  for  the  proposed  filter  design. 

•  Investigate  the  design  and  fabrication  of  nanowires  which  exhibit  polarization 
filtering  capabilities. 

•  Investigate  integration  of  CMOS  spectral  image  sensor  with  aluminum  nanowire 
polarization  filters.  Foveon  image  sensor  and  our  novel  custom  made  spectral 
image  sensor  will  be  integrated  with  polarization  filters. 

•  Investigate  application-based  algorithms  for  the  spectral-polarization  image 
sensor. 

•  Investigating  the  relationship  between  polarization  properties  of  light,  spectral 
information,  structure  and  shape  of  objects. 

•  Investigating  adaptive  processing  techniques  using  spectral  and  polarization 
information  of  light. 

•  Investigating  recovery  of  circularly  polarization  information  of  light  using 
nanostrucutres  in  conjunction  with  spectral  and  linearly  polarized  light. 

2.  MAJOR  ACCOMPLISHMENTS  AND  FINDINGS 

2.1,Realization  of  spectral-polarization  imaging  sensor 

Current  division-of-focal-plane  polarization  imaging  sensors  can  perceive  intensity  and 
polarization  in  real  time  with  high  spatial  resolution,  but  are  oblivious  to  spectral  information.  We  present 
the  design  of  such  a  sensor,  which  is  also  spectrally  selective  in  the  visible  regime.  We  describe  its 
extensive  spectropolarimetric  characterization.  The  sensor  has  a  pixel  pitch  of  5  gm  and  imaging  array  of 
X  by  Y  elements.  Each  element  comprises  spectrally  sensitive  vertically  stacked  photodetectors  integrated 
with  a  140  nm  pitch  nanowire  linear  polarizer.  The  sensor  has  SNR  of  45dB,  extinction  ratio  of  ~3.5,  QE 
of  12%,  and  linearity  error  of  1%  in  the  green  channel. 

We  have  designed  the  sensor  by  the  monolithic  integration  of  pixelated  nanowire  linear 
polarization  filters  with  a  spectrally  selective  imaging  array.  The  sensor  can  therefore  simultaneously 
acquire  spectral  and  polarization  information  in  a  scene  with  high  spatial  and  temporal  resolution. 
Furthermore,  both  spectral  and  polarization  information  is  co-registered  by  virtue  of  the  imaging  sensor 
architecture.  The  sensor  also  has  the  benefits  of  being  compact,  lightweight  and  robust.  The  spectral  and 
polarimetric  capabilities  of  the  integrated  sensor  have  been  thoroughly  assessed  using  a  carefully 
designed  optoelectronic  setup  and  are  presented  in  this  paper.  The  metrics  used  to  evaluate  the  sensor  are 
the  following:  quantum  efficiency  of  the  vertically  stacked  photodiodes,  linearity  of  the  pixel’s  output 
voltage,  and  signal- to-noise  ratio  (SNR). 

More  details  on  the  spectral-polarization  sensor  can  be  found  in  the  archived  publication. 

-  M.  Kulkami  and  V.  Gmev,  "Integrated  spectral-polarization  imaging  sensor  with  aluminum 
nanowire  polarization  filters,"  Optics  Express,  vol.  20,  pp.  22997-23012,  2012. 
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2,2,Fabrication  of  nanowire  pixelated  polarization  filters 

We  developed  a  procedure  for  fabricating  a  dual  layer  aluminum  nanowire  polarization  fdter 
suitable  for  division  of  focal  plane  polarization  imaging  sensors  for  the  visible  spectrum.  The  dual  layer 
micropolarization  array  is  fabricated  using  combination  of  optimized  interference  lithography  and 
microfabrication  steps  resulting  in  a  pixelated  polarization  filter  array  with  1 8  micron  pitch  and  extinction 
ratios  of  95  at  700nm  incident  light  wavelength.  The  filter  array  is  abutted  to  the  surface  of  a  custom 
CMOS  imaging  sensor  and  optical  test  are  performed  on  the  array. 

The  extinction  ratios  of  the  polarization  filter  arrays  are  evaluated  at  several  different 
wavelengths.  The  maximum  extinction  ratio  is  achieved  at  longer  wavelength,  i.e.  700nm,  and  is  95.  The 
minimum  extinction  ratio  is  achieved  at  shorter  wavelength,  i.e.  470nm,  and  is  51.  The  wavelength 
dependence  of  the  nanowire  polarization  filters  is  an  inherent  property  of  the  filters.  If  the  width  of  the 
aluminum  nanowires  is  decreased,  the  extinction  ratios  of  the  filter  array  can  be  improved  as  suggested  by 
theoretical  modeling. 

More  details  on  the  nano-fabrication  method  for  pixelated  nanowire  polarization  filters  can  be 
found  in  the  archived  publication. 

V.  Gruev,  "Fabrication  of  a  dual-layer  aluminum  nanowires  polarization  filter  array,"  Optics 
Express,  vol.  19,  pp.  24361-24369,  Nov  21  2011. 

2,3.Interpolation  algorithms  for  division  of  focal  plane  imagers 

We  have  developed  several  different  interpolation  algorithms  for  division  of  focal  plane 
polarization  sensors.  We  started  with  adapting  first  order  algorithms  to  our  sensor,  such  as  bilinear, 
bicubic  and  gradient  based  interpolation  algorithms.  We  later  developed  an  interpolation  algorithm  using 
statistical  methods. 

Image  interpolation  and  denoising  are  important  techniques  in  image  processing.  These  methods 
are  inherent  to  digital  image  acquisition  as  most  digital  cameras  are  composed  of  a  2D  grid  of 
heterogeneous  imaging  sensors.  Current  polarization  imaging  employ  four  different  pixelated  polarization 
filters,  commonly  referred  to  as  division  of  focal  plane  polarization  sensors.  The  sensors  capture  only 
partial  information  of  the  true  scene,  leading  to  a  loss  of  spatial  resolution  as  well  as  inaccuracy  of  the 
captured  polarization  information.  Interpolation  is  a  standard  technique  to  recover  the  missing 
information  and  increase  the  accuracy  of  the  captured  polarization  information.  Here  we  focus 
specifically  on  Gaussian  process  regression  as  a  way  to  perform  a  statistical  image  interpolation,  where 
estimates  of  sensor  noise  are  used  to  improve  the  accuracy  of  the  estimated  pixel  information.  We  further 
exploit  the  inherent  grid  structure  of  this  data  to  create  a  fast  exact  algorithm  that  operates  in  O 
□N3/2_(vs.  the  naive  O  N3_),  thus  making  the  Gaussian  process  method  computationally  tractable  for 
image  data.  This  modeling  advance  and  the  enabling  computational  advance  combine  to  produce 
significant  improvements  over  previously  published  interpolation  methods  for  polarimeters,  which  is 
most  pronounced  in  cases  of  low  signal-to-noise  ratio  (SNR).  We  provide  the  comprehensive 
mathematical  model  as  well  as  experimental  results  of  the  GP  interpolation  performance  for  division  of 
focal  plane  polarimeter. 

More  details  about  our  interpolation  methods  can  be  found  in  the  archived  publication: 

-  S.  K.  Gao  and  V.  Gruev,  "Bilinear  and  bicubic  interpolation  methods  for  division  of  focal  plane 
polarimeters,"  Optics  Express,  vol.  19,  pp.  26161-26173,  Dec  19  2011. 

S.  Gao  and  V.  Gruev,  "Gradient-based  interpolation  method  for  division-of- focal-plane 
polarimeters,"  Optics  Express,  vol.  21,  pp.  1137-1151,  2013. 


DISTRIBUTION  A:  Distribution  approved  for  pubiic  reiease 


-  E.  Gilboa,  J.  Cunningham,  A.  Nehorai  and  V.  Gruev,  “Image  interpolation  and  denoising  for 
division  of  focal  plane  sensors  using  Gaussian  Processes,”  Optics  Express,  vol.  22,  Issue  12,  pp.  15277- 
15291,2014. 

2,4,Calibration  of  division  of  focal  plane  sensors: 

We  have  developed  two  calibration  methods  for  division-of-focal-plane  polarimeters.  Typical 
division-of-focal-plane  polarimeters  for  the  visible  spectrum  employ  nanowires  in  order  to  construct 
linear  polarization  fdters.  Mismatches  in  the  size  of  the  nanowires  will  lead  to  optical  variations  at  the 
macro  scale  and  we  outline  two  calibration  methods  which  mitigate  these  effects.  Both  methods  were 
developed  from  the  same  linear  model  for  polarization  pixels,  but  one  treats  each  pixel  independently,  and 
the  other  treats  super-pixel  groups  together.  We  showed  that  the  super-pixel  approach  is  mathematically 
more  powerful  than  the  single-pixel  approach  and  can  correct  both  the  typical  photodetector  gain  and 
offset  non- idealities  in  addition  to  polarization  sensitive  flaws  such  as  non-ideal  filter  orientations  and 
non-ideal  filter  diattenuation  coefficients.  The  single-pixel  approach  can  only  correct  for  non-ideal  gains 
and  offsets. 

The  measurements  of  our  visible-spectrum  linear  DoFP  polarimeter  show  that  a  majority  of  the 
non-uniformity  between  pixels  is  in  their  gains  and  offsets,  but  a  significant  amount  of  variation  occurs  in 
the  model  parameters  that  the  single-pixel  approach  cannot  correct,  including  a  constant  rotational  offset 
and  moderate  variations  in  filter  diattenuations.  Thus  we  have  shown  that  calibrating  each  pixel 
independently  reduces  DoEP  reconstruction  errors  from  12%  to  10%  for  moderate  light  illuminations. 
Calibrating  each  super-pixel  as  a  unit  reduces  the  RMSE  to  approximately  1%.  Similar  reductions  in  error 
occur  for  reconstructing  the  intensity  and  AoP  images.  These  figures  indicate  that  the  super-pixel 
calibration  method  is  worth  the  extra  computational  effort,  but  there  are  still  some  un-addressed  sources 
of  error.  These  unaddressed  sources  of  error  include  the  image  sensor’s  non-linear  response,  temporal 
noise,  including  the  photon  shot  noise  and  readout  noise,  and  the  non-uniformities  in  the  fiat-field  that  the 
calibration  apparatus  produces. 

Finally,  we  showed  that  though  the  calibration  parameters  were  determined  using  a  tungsten- 
halogen  lamp  for  illumination  with  only  an  IR  blocking  filter  in  place,  they  performed  well  across  the 
visible  spectral  range  of  the  polarimeter.  It  is  also  worth  noting  that  the  optical  properties  of  the 
polarimeter  are  stable  enough  that  the  same  calibration  parameters  have  been  used  with  no  measurable 
difference  for  about  two  years  during  the  development  of  this  work.  The  improvements  in  the  quality  of 
real-life  images  obtained  from  a  division  of  focal  plane  polarimeter  for  the  visible  spectrum  after  applying 
the  two  calibration  methods  are  also  demonstrated  in  this  paper. 

More  details  about  our  calibration  methods  can  be  found  in  the  archived  publication: 

S.  Powell  and  V.  Gruev,  "Calibration  methods  for  division-of-foeal-plane 
polarimeters,"  Opties  Express,  vol.  21,  pp.  21040-21056,  2013. 

2,5,Opto-electronic  evaluation  of  division  of  focal  plane  polarization  imagers: 

The  development  of  high  resolution  division-of-focal-plane  polarimeters  in  the  visible  spectrum 
allows  real-time  capture  of  two  chief  properties  of  interest,  the  degree  of  linear  polarization  and  the  angle 
of  polarization.  The  accuracy  of  these  two  parameters  can  be  influenced  by  a  number  of  factors  in  the 
imaged  scene,  from  the  incident  intensity  and  wavelength,  to  the  lens  used  for  image  capture.  The 
alignment,  transmission,  and  contrast  ratios  of  the  pixel  matched  filters  also  impact  the  measured 
parameters.  A  system  of  measurements  are  presented  here  that  show  how  these  factors  can  determine  the 
quality  of  a  division-of-focal-plane  polarimeter. 

Visible  spectrum  DoFP  polarimeters  allow  the  estimation  of  the  DoLP  and  AoP  at  high  spatial 
and  temporal  resolution.  A  variety  of  factors,  including  the  intensity,  wavelength,  and  the  optical  path  of 
the  incident  light  influence  the  accuracy  of  this  estimation.  Presented  here  are  a  formal  system  of 


DISTRIBUTION  A:  Distribution  approved  for  pubiic  reiease 


experiments  designed  to  quantify  these  effects  for  a  DoFP  sensor.  The  experiments  give  a  complete 
picture  of  the  estimation  accuracy  of  the  DoLP  and  AoP.  They  illustrate  the  effects  that  mismatched 
analyzer  filters  have  on  the  accuracy  of  the  DoLP  and  AoP  estimation.  They  show  how  the  estimations 
are  functions  of  the  spectrum  of  incident  light,  performing  worse  for  shorter  wavelengths,  but  remaining 
relatively  constant  over  the  broader  visible  range.  They  also  demonstrate  how  important  collimation  is  to 
reducing  optical  crosstalk,  showing  that  light  divergence  is  a  key  limitation  to  DoLP  and  AoP  estimation. 
They  measured  that  this  degradation  from  divergence  directly  impacts  the  choice  of  lens,  with  narrower 
fields-of-view  contributing  less  error.  Finally,  they  show  that  the  sensor  is  less  accurate  for  lower 
intensities,  due  to  a  lowered  SNR.  The  experiments  presented  here  allow  for  an  objective  means  to 
evaluate  and  compare  DoFP  polarimeters. 

More  details  about  our  opto-electronic  evaluation  methods  can  be  found  in  the  archived 
publication: 

T.  York  and  V.  Gruev,  "Characterization  of  a  Visible  Speetrum  Division-of-Foeal- 
Plane  Polarimeter,"  Applied  Optics,  vol.  51,  pp.  5392-5400,  2012. 

2,6,Development  of  a  bio-inspired  polarization  camera  and  applications  to  detecting 

neural  activity: 

We  present  recent  work  on  bio-inspired  polarization  imaging  sensors  and  their  applications  in 
biomedicine.  In  particular,  we  focus  on  three  different  aspects  of  these  sensors.  First,  we  describe  the 
electro-optical  challenges  in  realizing  a  bio-inspired  polarization  imager,  and  in  particular,  we  provide  a 
detailed  description  of  a  recent  low-power  CMOS  polarization  imager.  Second,  we  focus  on  signal 
processing  algorithms  tailored  for  this  new  class  of  bio-inspired  polarization  imaging  sensors,  such  as 
calibration  and  interpolation.  Third,  the  emergence  of  these  sensors  has  enabled  rapid  progress  in 
characterizing  polarization  signals  and  environmental  parameters  in  nature,  as  well  as  several  biomedical 
areas,  such  as  label-free  optical  neural  recording,  dynamic  tissue  strength  analysis,  and  early  diagnosis  of 
flat  cancerous  lesions  in  a  murine  colorectal  tumor  model.  We  highlight  results  obtained  from  these  three 
areas  and  discuss  future  applications  for  these  sensors. 

More  details  about  our  bio-inspired  polarization  sensor  and  its  application  to  biomedical 
applications  can  be  found  in  the  archived  publication: 

-  T.  Charanya,  T.  York,  S.  Bloch,  G.  Sudlow,  R.  Tang,  W.  Akers,  D.  Rubin,  V.  Gruev,  S. 
Achilefii,  “Trimodal  color-fluorescence-polarization  endoscopy  aided  by  a  tumor  selective  molecular 
probe  accurately  detects  flat  lesions  in  colitis-associated  cancer,”  Journal  of  Biomedical  Optics,  2014. 

-  T.  York,  S.  Powell,  S.  Gao,  L.  Kahan,  T.  Charanya,  D.  Saha,  N.  Roberts,  T.  Cronin,  J.  Marshall, 
S.  Achilefii,  S.  Lake,  B.  Raman,  and  V.  Gruev,  “Bio-Inspired  Polarization  Imaging  Sensors:  From 
Circuits  and  Optics  to  Signal  Processing  Algorithms  and  Biomedical  Applications,”  Proceedings  of  the 
IEEE,  Vol.  102,  No.  10,  pp.  1450-1469,  2014. 

-  T.  York,  E.  Kahan,  S.  Eake  and  V.  Gruev,  “Real-time  high-resolution  measurement  of  collagen 
alignment  in  dynamically  loaded  soft  tissue,”  Journal  of  Biomedical  Optics,  vol.  19,  no.  6,  pp.  066011, 
2014. 


-  Y.  Eiu,  T.  York,  W.  Akersa,  G.  Sudlowa,  V.  Gruev,  and  S.  Achilefua,  "Complementary 
Fluorescence-Polarization  Microscopy  using  Division-of-Focal-Plane  Polarization  Imaging  Sensor," 
Journal  of  Biomedical  Optics,  2012. 

2,7,  Surface  normal  extraction  using  bio-inspired  polarization  sensor 

The  polarization  properties  of  reflected  light  capture  important  information  about  the  object’s 
inherent  properties:  material  composition,  i.e.  index  of  refraction  and  scattering  properties,  and  shape  of 
the  object,  i.e.  surface  normal.  Polarization  information  therefore  has  been  used  for  surface  reconstruction 
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using  a  single-view  camera  with  unpolarized  incident  light.  However,  this  surface  normal  reconstruction 
technique  suffers  from  a  zenith  angle  ambiguity.  In  this  paper,  we  have  utilized  circularly  polarized  light 
to  solve  for  the  zenith  ambiguity  by  developing  a  detailed  model  using  Mueller  matrix  formulism  and 
division  of  focal  plane  polarization  imaging  technology.  Experiment  results  validate  our  model  for 
accurate  surface  reconstruction. 

The  contribution  of  this  work  is  to  exploit  the  phase  information  of  light,  i.e.  circular  polarization 
properties  of  light,  to  determine  shape.  The  phase  difference  between  the  two  orthogonal  components  of 
the  light  wave  determines  the  elliptical  component  of  the  polarization  state.  Using  the  elliptical 
component  of  polarization,  the  ambiguity  of  the  zenith  angle  in  Fresnel’s  reflections  can  be  resolved  from 
a  single  viewpoint  using  a  single  imaging  sensor.  In  addition  this  work  applies  the  powerful  concept  of 
Stokes  vectors  and  Mueller  matrices  to  analyze  more  complex  input  light  sources  and  derive  a  closed- 
form  solution  for  both  the  zenith  and  azimuth  angles.  To  solve  for  the  azimuth  ambiguity,  this  paper  uses 
a  reconstruction  method  based  on  the  one  presented  in  the  literature.  Furthermore,  this  paper  employs  a 
division  of  focal  plane  polarimeter  capable  of  acquiring  polarization  information  at  every  imaged  frame 
with  high  spatial  resolution.  The  imaging  sensor  is  realized  by  monolithically  integrating  pixelated 
nanowire  polarization  filters  with  an  array  of  pixels  that  use  CCD  technology. 

More  details  about  our  3-D  surface  normal  extraction  using  polarization  information  can  be  found 
in  the  archived  publication: 

-  N.  Missael  Garcia,  I.  de  Erausquin,  C.  Edmiston,  and  V.  Gruev,  “Surface  normal  reconstruction 
using  circularly  polarized  light,”  Optics  Express,  2015. 

2,8,Dynamic  Polarization  Signaling  in  Swordtails  Alters  Female  Mate  Preference 

Polarization  of  light  — and  visual  sensitivity  to  it — is  pervasive  across  aquatic  and  terrestrial 
environments.  Documentation  of  invertebrate  use  of  polarized  light  is  widespread  from  navigation  and 
foraging  to  mate-recognition.  In  contrast,  evidence  for  vertebrate  use  of  polarized  light  is  limited,  and 
accounts  of  polarization  communication  by  vertebrates  are  conspicuously  missing.  Here  we  investigate 
polarization-mediated  communication  by  northern  swordtails,  Xiphophorus  nigrensis,  using  a  custom- 
built  videopolarimeter  to  measure  dynamic  polarization  signals  and  a  novel  experimental  paradigm  that 
manipulates  polarization  signals  without  modifying  their  brightness  or  color.  We  conducted  mate  choice 
trials  in  an  experimental  tank  that  illuminates  a  pair  of  males  with  light  passed  through  a  polarization  and 
a  diffusion  filter.  By  alternating  the  order  of  these  filters  between  males,  we  presented  females  with  live 
males  that  differed  in  polarization  features  by  >  100%,  but  with  intensity  and  color  differences  below 
detection  thresholds  (<5%).  Combining  videopolarimetry  and  polarization-manipulated  mate  choice 
trials,  we  found  sexually  dimorphic  polarized  reflectance,  polarization-dependent  mate  choice  behavior, 
and  differential  polarization  signaling  across  social  contexts.  Male  swordtails  exhibit  greater  polarization 
contrast  than  females,  and  females  preferentially  associate  with  high  polarization-reflecting  males.  Males 
also  adjust  polarization  signals  based  on  social  context  by  increasing  within-body  contrasts  and 
decreasing  body-to-background  contrasts  in  social  conditions  relative  to  asocial  conditions.  Polarization 
cues  in  mate  choice  contexts  may  provide  aquatic  vertebrates  with  enhanced  detection  of  specific  display 
features  (e.g.,  movements  or  angular  information),  as  well  as  a  dynamic  signaling  mechanism  that  may 
enhance  detection  by  intended  viewers  while  minimizing  detection  by  others. 

More  details  about  our  behavioral  work  using  polarization  information  can  be  found  in  the 
archived  publication: 

-  G.  Calabrese,  P.  Brady,  V.  Gruev,  and  M.  Gumming,  “Polarization  Signaling  in  Swordtails 
Alters  Female  Mate  Preference,”  Proceedings  of  National  Academy  of  Sciences,  Vol.  Ill,  No.  37,  pp. 
13397-13402,  2014. 
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4.  Investigate  integration  of  CMOS  spectral  image  sensor  with  aluminum  nanowire  polarization  filters. 

Foveon  image  sensor  and  our  novel  custom  made  spectral  image  sensor  will  be  integrated  with 
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with  spectral  and  linearly  polarized  light. 

I  believe  that  the  proposed  research  will  not  only  contribute  to  advancement  of  spectral  and  polarization 
imaging  but  also  make  significant  advances  in  novel  imaging  architectures,  integration  of  nanomaterials 
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